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ABSTRACT 

Currently,  there  is  no  standard  quantitative  measure  of  the  performance  of  an  image-based  tracker  algorithm.  It  is 
usually  described  as  remaining  in  track  for  the  entire  run  or  losing  track  at  some  point  during  a run.  This  is  purely  a 
qualitative  evaluation.  Without  a quantitative  measure  it  becomes  difficult  to  accurately  evaluate  the  performance  of  a 
tracker  algorithm.  We  have  developed  the  Tracker  Performance  Metric  (TPM)  specifically  for  this  purpose.  It  was 
designed  to  measure  the  output  performance,  on  a frame-by-frame  basis,  using  its  output  position  and  quality  (sometimes 
referred  to  as  confidence)  state.  The  TPM  can  also  be  used  as  a measure  of  algorithm  performance  to  compare  against 
the  Trackability  Metric.  The  Trackability  Metric  was  developed  by  AMCOM  to  determine  how  “trackable”  a set  of  data 
should  be.  The  TPM  will  be  described  and  results  presented. 

Keywords:  Tracker  Performance  Metric,  Aimpoint  Selection  Metric,  Quality  Measure 

INTRODUCTION 

The  purpose  of  this  paper  is  to  introduce  a family  of  functions  designed  to  yield  standard  quantitative  calculations  to  rate 
performance  for  image-based  tracker  and  aimpoint  selection  algorithms.  These  functions  are  adaptable  to  the  algorithm 
evaluator’s  desired  output  requirements.  They  also  have  the  ability  to  exploit  composite  tracker  and  aimpoint  algorithm 
technology  that  may  be  used  to  leverage  multiple  algorithms  within  a solution  suite.  This  approach  requires  a 
component  track  quality  measure  that  represents  the  algorithm's  inferred  performance.  We  believe  that  a quality  metric 
is  an  extremely  important  output  of  any  tracker  algorithm  for  producing  robust  solutions  for  today’s  military 
environment.  An  algorithm’s  ability  to  accurately  assess  its  current  quality  state  is  a crucial  metric  for  confidence 
measures  that  enable  internal  corrections  and  aid  in  externa!  guidance  decisions.  If  an  evaluator  does  not  have  the  ability 
to  measure  the  quality  of  his  tracker  algorithm,  the  quality  factor  can  be  input  as  the  maximum  value  of  one.  However, 
we  must  stress  that  this  would  not  be  considered  an  optimal  solution.  An  accurate  quality  measure  provides  the  most 
effective  representation  of  tracker  performance. 

METRIC  FORMULATION 

At  the  center  of  this  metric  concept  is  the  proposed  function  / (x,  y;  Q,  xc  ,ye),  where  {x,y)  are  image  coordinate 
variables,  Q is  the  algorithm’s  internally  assessed  quality  metric  (constrained  by  0 < Q < 1 ) and  (x0ly0)  is  the  algorithm’s 
position  output.  The  function  is  assumed  to  be  integrable  over  all  space,  “centered”  at  the  algorithm  output  position 
(x0,y0),  and  normalized  to  unit  area  ( Jj"  f{x,y,Q,xo,yo)  = 1 ).  The  metric  is  calculated  by  integrating  this  function  over 

all  space 

the  desired  area  of  interest  (An),  typically  centered  on  the  true  target  location  (xT,yi).  In  mathematical  form,  the 
performance  metric  is  represented  by  TPM  = Jj/(x, y, Q,  xo , yc ) . Because  of  the  normalization  criteria,  the  TPM  is 

Ab 

guaranteed  to  produce  a value  between  zero  and  one  ( 0 < TPM  < 1 ). 

The  TPM  was  developed  to  be  used  in  conjunction  with  the  Trackability  Metric  (TM)  or  Gray  Level  Co-occurrence 
Matrix  (GLCM).  For  detailed  information  on  the  TM,  we  refer  the  reader  to  [1,2,3].  The  Trackability  Metric  measures 
how  well  a tracker  should  perform  based  on  the  tracker’s  frame-by-frame  input  imagery.  On  the  other  hand,  the  Tracker 
Performance  Metric  measures  how  well  a tracker  actually  does  perform  on  a frame-by-frame  basis.  Because  both 
functions  are  normalized  to  the  same  scale  (0-1),  they  are  easily  comparable.  In  general,  a robust  tracker  would  be 
expected  to  perform  equal  to  or  better  than  the  GLCM  representation. 
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To  illustrate  the  utility  of  this  proposed  metric,  it  is  simpler  if  we  select  a typical  example  of  a set  of  desired  output 
requirements  for  an  algorithm.  We  will  then  be  able  to  choose  a representative  for  the  function  f{x,y,Q,xo,y\)  and  the 

appropriate  area  of  interest  for  integration.  A typical  performance  goal  for  an  image-based  tracker  is  to  reference  all 
outputs  to  the  centroid  of  the  target  under  track.  Large  deviations  of  the  tracker  output  from  this  centroid  should  be 
penalized  as  well  as  incorrect  internal  assessments  of  the  tracker’s  quality  measure.  We  have  selected  the  area  of  interest 
as  the  entire  target  area  but  internal  regions  within  the  target  itself  are  also  viable  candidates  (especially  for  aimpoint 
selection  evaluations).  To  review,  we  want  to  penalize  tracker  output  deviations  from  the  centroid  of  the  target  and  also 
incorrect  internal  assessments  produced  from  the  tracker’s  quality  metric  evaluation. 


In  order  to  demonstrate  this  concept,  choose  the  desired  area  of  interest,  AD,  to  be  a rectangular  shaped  target  truth  box 
having  dimensions  (Lx,Lt)  that  tightly  encompass  the  target  and  is  centered  on  the  target  centroid.  Note  that  this  region 
does  not  have  to  be  rectangular.  It  could  also  be  chosen  to  reflect  the  true  target  segment.  As  we  will  show  later, 
choosing  a rectangular  region  simplifies  the  computation.  For  our  function,  we  choose  a 2-dimensional  multivariate 

Gaussian  (normal)  distribution,  , centered  at  the  tracker’s  output  position  // = 

L y0 


and  0 < Q < 1 is  the  confidence  measure  output  of  the  tracker.  The  choice 

yj 


of  the  diagonal  elements  of  2 comes  from  our  desire  to  have  at  least  99%  of  the  area  under  the  Gaussian  surface 

L l 

contained  within  the  target  truth  box  when  the  Q value  equals  one  (— i - = 3cr;— ~ = 3<r).  This  gives  the  following 

2 Q 2 Q 

functional  form  for  N (^r ; : 


The  TPM  is  calculated  by  integrating  this  normal  distribution  only  over  the  portion  of  the  image  that  the  target  occupies. 
For  a rectangular  truthed  region,  AD,  this  is  represented  by  the  following  equation: 
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where  Lx  = x,  - xt  > 0 , Lv  = )\  - )\  > 0 and  erf(x)  = — =■  Jexp(-r)i#  is  the  standard  error  function  used  by 

\7T  „ 

Matlab  (erf.m). 


The  crucial  element  in  this  formulation  is  that  the  function  should  be  “centered”  around  the  algorithm’s  output  track 
point  but  the  integration  should  only  be  taken  over  the  desired  area  of  interest.  A mixture  model  of  Gaussians  matched  to 
predefined  lethality  maps  could  be  used  to  effectively  quantify  tracker  performance  in  a more  sophisticated  aimpoint 
selection  evaluation  mode.  As  the  output  drifts  away  from  the  desired  area  or  the  tracker’s  quality  metric  becomes 
inaccurate,  the  area  under  f(x,y\Q,xo,yc)  will  move  away  or  expand  into  or  out  of  region  AD. 


EXAMPLES 

Figure  ] illustrates  TPM  results  for  a tracker  on  US  Army  Aviation  and  Missile  Command  (AMCOM)  data.  The  images 
show  the  truth  bounding  box  and  the  1 -sigma,  2-sigma  and  3-sigma  ellipses  of  the  normal  distribution.  In  the  left  image, 
the  TPM  = 0.97  because  the  tracker’s  output  is  located  near  the  center  of  the  target  and  the  tracker  has  a high  confidence 
value.  The  right  image  has  a TPM  = 0.88 1 , even  though  the  output  is  located  near  the  center  of  the  target.  This  is  because 
the  tracker’s  quality  (Q  or  TQ)  is  equal  to  0.707.  This  has  allowed  more  of  the  area  to  bleed  outside  of  AD.  By  design,  a 
tracker  producing  an  output  on  the  target  centroid,  having  a high  degree  of  confidence  (TQ  = 1)  will  give  a TPM 
approximately  equal  to  one. 
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Tracker  Performance  Metric 


■x.,y,  - Tracker  output 
position 

j Q — Quality  measure 
produced  by  tracker 
(0<^'t) 

,L  - Target  horizontal 
-length 

L = Target  vertical 
length 


Frame  1145  Fivnr.:TQ=  i.ooo.tpm=  0.814  Frame  1885  jtctd : to  = 0.707. tpm  = 0 as  l im=  d.736 


Figure  1:  Tracker  Performance  Metric  example  1. 


Figure  2 illustrates  TPM  results  for  another  tracker  run  on  a different  data  set.  Note  the  right  image.  The  TPM  has  the 
very  low  value  of  0.088.  Not  only  has  the  tracker  produced  an  output  on  this  frame  far  from  the  target  centroid,  but  also 
it  is  claiming  a relatively  high  confidence  of  being  on  track  (0.739).  This  is  a problematic  result  for  the  tracker.  The 
tracker  is  claiming  good  performance  but  actually  producing  poor  results.  Note  that  had  the  tracker  been  reliable  enough 
to  claim  a poor  quality  rating,  the  ellipses  would  have  expanded  outward  from  the  tracker  output  and  towards  the  desired 
region  of  interest  (the  target).  This  would  have  yielded  more  area  over  the  target  region,  and  therefore,  a higher  TPM 
value.  This  result  is  fundamental  to  our  metric  formulation.  Because  the  tracker  has  a better  understanding  of  its 
performance  limitations,  it  is  more  reliable  and  the  algorithm  performance  metric  reflects  it.  If  the  tracker  can  accurately 
identify  when  it  has  a lower  confidence  or  quality,  it  can  potentially  indicate  when  to  go' into  track  or  when  not  to  update 
temporal  information.  A tracker  that  makes  good  assessments  regarding  its  internal  state,  yields  more  accurate  guidance 
decisions,  and  therefore,  more  accurate  system  results. 

Figure  3 shows  an  example  of  the  Tracker  Performance  Metric  plotted  against  the  Trackability  Metric.  It  can  be  seen 
that  the  TPM  corresponds  well  with  the  GLCM  TM. 
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The  TPM  is'  lowest 
in.  frame  750. 

, Notice  that  on  that 
frame  the  tracker  is 
tracking  the  growl 
just  behind  the 
target  with  a high 
quality  value  . 


Flame  315  OfcapE2:TQ  * 1000,  IPM«  omiM=  0.95C 


Frame  750  0fcm.pi:2:T0  = 0.739, TPM=  0.083. TW=0.8SO 


50  105  L50  200  250 


Figure  2:  Tracker  Performance  Metric  example  2. 


Frame  # 


Figure  3:  Tracker  Performance  versus  Trackability  Metric  example. 
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SUMMARY 

This  paper  has  derived  a quantitative  method  for  evaluating  the  performance  of  any  image-based  tracker  algorithm.  This 
method,  known  as  the  Trackability  Performance  Metric,  can  be  used  to  compare  the  performance  of  two  or  more 
independent  tracker  or  aimpoint  algorithms.  It  can  also  be  used  in  conjunction  with  the  Gray  Level  Co-occurrence 
Matrix  Trackability  Metric  to  determine  how  well  a given  algorithm  performs  verses  how  well  it  should  perform  on  a 
given  set  of  data. 
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